Adjusting camera exposure for three-dimensional depth sensing and two-dimensional imaging

ABSTRACT

An example method includes setting an exposure time of a camera of a distance sensor to a first value, instructing the camera to acquire a first image of an object in a field of view of the camera, where the first image is acquired while the exposure time is set to the first value, instructing a pattern projector of the distance sensor to project a pattern of light onto the object, setting the exposure time of the camera to a second value that is different than the first value, and instructing the camera to acquire a second image of the object, where the second image includes the pattern of light, and where the second image is acquired while the exposure time is set to the second value.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/353,859, filed Mar. 14, 2019, which in turn claims the priority ofU.S. Provisional Patent Application Ser. No. 62/645,190, filed Mar. 20,2018. Both of these applications are herein incorporated by reference intheir entireties.

BACKGROUND

U.S. patent application Ser. Nos. 14/920,246, 15/149,323, and 15/149,429describe various configurations of distance sensors. Such distancesensors may be useful in a variety of applications, including security,gaming, control of unmanned vehicles, and other applications.

The distance sensors described in these applications include projectionsystems (e.g., comprising lasers, diffractive optical elements, and/orother cooperating components) which project beams of light in awavelength that is substantially invisible to the human eye (e.g.,infrared) into a field of view. The beams of light spread out to createa pattern (of dots, dashes, or other artifacts) that can be detected byan appropriate light receiving system (e.g., lens, image capturingdevice, and/or other components). When the pattern is incident upon anobject in the field of view, the distance from the sensor to the objectcan be calculated based on the appearance of the pattern (e.g., thepositional relationships of the dots, dashes, or other artifacts) in oneor more images of the field of view, which may be captured by thesensor's light receiving system. The shape and dimensions of the objectcan also be determined.

For instance, the appearance of the pattern may change with the distanceto the object. As an example, if the pattern comprises a pattern ofdots, the dots may appear closer to each other when the object is closerto the sensor, and may appear further away from each other when theobject is further away from the sensor.

SUMMARY

An example method includes setting an exposure time of a camera of adistance sensor to a first value, instructing the camera to acquire afirst image of an object in a field of view of the camera, where thefirst image is acquired while the exposure time is set to the firstvalue, instructing a pattern projector of the distance sensor to projecta pattern of light onto the object, setting the exposure time of thecamera to a second value that is different than the first value, andinstructing the camera to acquire a second image of the object, wherethe second image includes the pattern of light, and where the secondimage is acquired while the exposure time is set to the second value.

In another example, a non-transitory machine-readable storage medium isencoded with instructions executable by a processor. When executed, theinstructions cause the processor to perform operations including settingan exposure time of a camera of a distance sensor to a first value,instructing the camera to acquire a first image of an object in a fieldof view of the camera, where the first image is acquired while theexposure time is set to the first value, instructing a pattern projectorof the distance sensor to project a pattern of light onto the object,setting the exposure time of the camera to a second value that isdifferent than the first value, and instructing the camera to acquire asecond image of the object, where the second image includes the patternof light, and where the second image is acquired while the exposure timeis set to the second value.

In another example, a distance sensor includes a pattern projectorconfigured to project a pattern of light onto an object, a camera, acontroller configured to set an exposure time of the camera to a firstvalue when the pattern projector is not projecting the pattern of lightonto the object and to set the exposure time of the camera to a secondvalue when the pattern projector is projecting the pattern of light ontothe object, and a processor configured to calculate a distance from thedistance sensor to the object based on a first image captured when theexposure time is set to the first value and a second image captured whenthe exposure time is set to the second value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example distance sensor of thepresent disclosure;

FIG. 2 is a flow diagram illustrating one example of a method foradjusting the camera exposure of a distance sensor for three-dimensionaldepth sensing and two-dimensional image capture, according to thepresent disclosure;

FIG. 3 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera, and thedistance projection for three-dimensional distance measurement, where asingle light source emits light during separate exposures forthree-dimensional distance information and two-dimensional imageacquisition, and where three-dimensional distance measurement andtwo-dimensional image acquisition alternate every other frame;

FIG. 4 is a block diagram illustrating an example distance sensor of thepresent disclosure;

FIG. 5 is a flow diagram illustrating one example of a method foradjusting the camera exposure of a distance sensor for three-dimensionaldepth sensing and two-dimensional image capture, according to thepresent disclosure;

FIG. 6 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera, the distanceprojection for three-dimensional distance measurement, and the lightemission for two-dimensional image acquisition, where a first lightsource emits light at or near the time of three-dimensional dataacquisition and second, separate light source emits light at the time oftwo-dimensional image acquisition, and three-dimensional distancemeasurement and two-dimensional image acquisition alternate every otherframe;

FIG. 7 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera, the distanceprojection for three-dimensional distance measurement, and the lightemission for two-dimensional image acquisition, where a first lightsource emits light at or near the time of three-dimensional dataacquisition and second, separate light source emits light at the time oftwo-dimensional image acquisition, and three-dimensional distancemeasurement and two-dimensional image acquisition alternate everypredetermined number of frames;

FIG. 8 is a block diagram illustrating an example distance sensor of thepresent disclosure;

FIG. 9 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera (e.g., a videocamera), the distance projection for three-dimensional distancemeasurement, and the light emission for two-dimensional imageacquisition, where two light projection systems (e.g., used forthree-dimensional distance data acquisition) are used and the exposuredurations for three-dimensional data acquisition and two-dimensionalimage capture are the same;

FIG. 10 is a flow diagram illustrating one example of a method foradjusting the camera exposure of a distance sensor for three-dimensionaldepth sensing and two-dimensional image capture, according to thepresent disclosure;

FIG. 11 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera (e.g., a videocamera), the distance projection for three-dimensional distancemeasurement, and the light emission for two-dimensional imageacquisition, where information about shutter speed at the time oftwo-dimensional image acquisition is fed back to the timing forthree-dimensional distance data acquisition; and

FIG. 12 depicts a high-level block diagram of an example electronicdevice for calculating the distance from a sensor to an object.

DETAILED DESCRIPTION

The present disclosure broadly describes an apparatus, method, andnon-transitory computer-readable medium for adjusting the cameraexposure of a distance sensor for three-dimensional depth sensing andtwo-dimensional image capture. As discussed above, distance sensors suchas those described in U.S. patent application Ser. Nos. 14/920,246,15/149,323, and 15/149,429 determine the distance to an object (and,potentially, the shape and dimensions of the object) by projecting beamsof light that spread out to create a pattern (e.g., of dots, dashes, orother artifacts) in a field of view that includes the object. The beamsof light may be projected from one or more laser light sources whichemit light of a wavelength that is substantially invisible to the humaneye, but which is visible to an appropriate detector (e.g., of the lightreceiving system). The three-dimensional distance to the object may thenbe calculated based on the appearance of the pattern to the detector.

In some cases, a two-dimensional image of the object may also becaptured (e.g., by a camera of the light receiving system) and used toimprove the three-dimensional distance measurement. For example, areference mark may be affixed to the object. Then, when the availableamount of three-dimensional information (e.g., number of dots of thepattern) is insufficient for making an accurate distance measurement,information from a two-dimensional image of the object (including thereference mark) may be used to supplement the three-dimensionalinformation. It may also be possible to determine environmentalcharacteristics such as external brightness, object reflectance, and thelike from the two-dimensional image. This information may be used toadjust the projected beams (and, consequently, the projected pattern) toimprove the three-dimensional distance measurement.

Various factors, however, make it difficult to acquire three-dimensionalobject data (e.g., including pattern data) and a two-dimensional objectimage in quick succession with the same camera. For instance, noiseintroduced by ambient light may make it difficult for the detector toclearly detect the pattern formed by the beams. One way to mitigate theeffects of ambient light is to incorporate a narrow band-pass filter inthe light receiving system of the sensor, e.g., where the filter allowsonly infrared light to pass. However, if the amount of ambient light isvery great (such as might be the case outdoors), then the difference inbrightness between the pattern and the ambient light may be very small.Moreover, if the exposure time of the light receiving system is not setappropriately, then the relationship between the exposure value and thesensor latitude may cause unwanted saturation. In either case, it maystill be difficult for the detector to distinguish the pattern formed bythe beams from the ambient light, even if a narrow band-pass filter isused. For example, when both the image of the pattern formed by thebeams and the ambient light exceed the sensor latitude, saturation mayoccur. However, an image of the pattern may become clearer by reducingthe exposure time so that the amount of light input to the lightreceiving system is within the range of the sensor's latitude.

Alternatively or in addition, when the amount of ambient light is great,the pattern may be easier for the detector to distinguish if thebrightness of the beams that form the pattern is increased relative tothe brightness of the ambient light. However, from a safety perspective,increasing the brightness of the beams may come with some risk, asexposure to the brighter beams may be harmful to the human eye. Thus,the emission time of the lasers may be shortened to minimize the risk,and the exposure time of the light receiving system may also beshortened to reduce ambient light.

Although increasing the brightness of the pattern and reducing theexposure time of the light receiving system may improve the detector'sability to acquire three-dimensional information, these modificationsmay also impair the camera's ability to capture a useful two-dimensionalimage. For instance, a two-dimensional image that is captured under ashortened exposure time is likely to be dark. Generally, a longerexposure time may be needed to capture a clearer two-dimensional image.

Thus, in summary, the optimal camera exposure time for detecting athree-dimensional projection pattern and the optimal camera exposuretime for capturing a two-dimensional image may be very different. Thismakes it difficult to detect the three-dimensional projection patternand to capture the two-dimensional image simultaneously, or within arelatively short period of time (e.g., less than one second), with thesame camera.

Examples of the present disclosure provide a distance sensor that iscapable of performing three-dimensional information acquisition (e.g.,from a pattern of projected light) and two-dimensional image acquisitionin quick succession, with a single camera. In one example, the lightsource used to provide illumination for the two-dimensional imageacquisition has the same wavelength as the light source that is used toproject the pattern for three-dimensional information acquisition. Thiseliminates the need for a band-pass filter in the light receiving systemof the distance sensor.

FIG. 1 is a block diagram illustrating an example distance sensor 100 ofthe present disclosure. The distance sensor 100 may be used to detectthe distance d to an object 114. In one example, the distance sensor 100shares many components of the distance sensors described in U.S. patentapplication Ser. Nos. 14/920,246, 15/149,323, and 15/149,429. Forinstance, in one example, the distance sensor comprises a camera (orother image capturing device) 102, a processor 104, a controller 106,and a pattern projector 108.

In one example, the camera 102 may be a still or video camera. Thecamera 102 may be capable of capturing three-dimensional distance data.For instance, the camera 102 may include a detector that is capable ofdetecting a pattern of light that is projected onto the object 114,where the projected light is of a wavelength that is substantiallyinvisible to the human eye (e.g., infrared). The camera 102 may also becapable of capturing two-dimensional red, green, blue (RGB) images ofthe object 114. Thus, in one example, the camera 102 may be a red,green, blue infrared (RGBIR) camera. In this case, infrared lightemitted for three-dimensional distance sensing may be input only to thepixels of the camera 102 with the IR filter, while other wavelengths oflight can be recognized as color images by the pixel(s) on the RGBfilter. Thus, the detector of the camera can detect red, green, blue andinfrared simultaneously, can detect only infrared, or can detect onlyred, green, and blue. Because the three-dimensional distance sensingdepends on the intensity of the projected pattern of light, and thetwo-dimensional imaging depends on external brightness, the optimalexposure time for the IR and RGB portions of the camera 102 will bedifferent. The camera 102 may have a fish-eye lens, and may beconfigured to capture image data of a field of view of up to 180degrees.

The camera 102 may send captured image data to the processor 104. Theprocessor 104 may be configured to process the captured image data(e.g., three-dimensional distance data and two-dimensional image data)in order to calculate the distance to the object 114. For instance, thedistance may be calculated in accordance with the methods described inU.S. patent application Ser. Nos. 14/920,246, 15/149,323, and15/149,429.

The controller 106 may be configured to control operation of the othercomponents of the distance sensor, e.g., the operations of the camera102, the processor 104, and the pattern projector 108. For instance, thecontroller 106 may control the exposure time of the camera 102 (e.g.,the duration for which the camera's shutter is open), and the timingwith which the camera 102 captures images (including images of theobject 114). As discussed in further detail below, the controller 106may set two separate exposure durations for the camera 102: a firstexposure duration during which an image of the object 114 is captured atthe same time that the pattern projector 108 projects a pattern onto theobject 114 (e.g., for three-dimensional distance sensing), and a secondexposure duration during which an image of the object 114 is captured ata time when the pattern projector 108 does not project a pattern ontothe object 114 (e.g., for two-dimensional image acquisition). In oneexample, the controller 106 may alternate between the first exposureduration and the second exposure duration.

The controller 106 may also control the duration for which the patternprojector 108 projects the pattern of light onto the object 114, as wellas the timing with which the pattern projector 108 projects the patternof light onto the object 114. For instance, the controller 106 maycontrol the duration of pulses emitted by a light source of the patternprojector 108, as discussed in further detail below.

The pattern projector 108 may comprise various optics configured toproject the pattern of light onto the object 114. For instance, thepattern projector 108 may include a laser light source, such as avertical cavity surface emitting laser (VCSEL) 110 and a diffractiveoptical element (DOE) 112. The VCSEL 110 may be configured to emit beamsof laser light under the direction of the controller 106 (e.g., wherethe controller 106 controls the duration of the laser pulses). The DOE112 may be configured to split the beam of light projected by the VCSEL110 into a plurality of beams of light. The plurality of beams of lightmay fan or spread out, so that each beam creates a distinct point (e.g.,dot, dash, x, or the like) of light in the camera's field of view.Collectively, the distinct points of light created by the plurality ofbeams form a pattern. The distance to the object 114 may be calculatedbased on the appearance of the pattern on the object 114.

FIG. 2 is a flow diagram illustrating one example of a method 200 foradjusting the camera exposure of a distance sensor for three-dimensionaldepth sensing and two-dimensional image capture, according to thepresent disclosure. The method 200 may be performed, for example, by theprocessor 104 illustrated in FIG. 1. For the sake of example, the method200 is described as being performed by a processing system.

The method 200 may begin in step 202. In step 204, the processing systemmay set the exposure time of a camera of a distance sensor to a firstvalue. The first value may define a duration of the exposure (e.g., afirst window of time for which the shutter of the camera is open toacquire image data).

In step 206, the processing system may instruct the camera to acquire afirst image of an object in the distance sensor's field of view. In oneexample, the first image is a two dimensional image (which includes nodata from projected patterns of light). In one example, the time ofexposure for the acquisition of the first image is therefore equal tothe first value.

In step 208, the processing system may instruct a pattern projector(e.g., a system of optics including a laser light source and diffractiveoptical element) of the distance sensor to project a pattern of lightonto the object. In one example, the pattern of light may comprise lightthat is emitted in a wavelength that is substantially invisible to thehuman eye (e.g., infrared). In one example, the instructions sent to thepattern projector may include instructions regarding when to startprojecting the pattern of light and for how long to project the patternof light (e.g., the timing and duration of laser pulses).

In step 210, the processing system may set the exposure time of thecamera to a second value. The second value may define a duration of theexposure (e.g., a second window of time for which the shutter of thecamera is open to acquire image data). In one example, the second valueis smaller than the first value.

In step 212, the processing system may instruct the camera to acquire asecond image of the object, where the second image also includes thepattern of light projected onto the object by the pattern projector. Inone example, the time of exposure for the acquisition of the secondimage is therefore equal to the second value.

In step 214 the processing system may instruct the pattern projector tostop projecting the pattern of light onto the object. For instance, theinstructions sent to the pattern projector may instruct the patternprojector to turn off a laser.

In step 216, the processing system may determine whether to stop imagingthe object. For instance, imaging of the object may stop if sufficientdata (e.g., from the first and second images) has been acquired tocalculate the distance to the object. If the processing system concludesin step 216 that imaging should not be stopped, then the method 200 mayreturn to step 204 and proceed as described above to capture additionalimages of the object.

Alternatively, if the processing system concludes in step 216 thatimaging should be stopped, then the method 200 may proceed to step 218.In step 218, the processing system may process the first and secondimages in order to determine the distance to the object. For instance,any of the methods described in in U.S. patent application Ser. Nos.14/920,246, 15/149,323, and 15/149,429 may be used to calculate thedistance. Alternatively, the processing system may send the first andsecond images to a remote processing system for the distancecalculation.

The method 200 may end in step 220.

FIG. 3 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera, and thedistance projection for three-dimensional distance measurement, where asingle light source emits light during separate exposures forthree-dimensional distance information and two-dimensional imageacquisition, and where three-dimensional distance measurement andtwo-dimensional image acquisition alternate every other frame. Forinstance, the timing chart of FIG. 3 may illustrate the operations ofthe distance sensor 100 of FIG. 1.

In particular, FIG. 3 shows seven frames, f₁-f₇, of the timing chart. Inone example, a three-dimensional distance measurement and atwo-dimensional image acquisition are performed alternately, every otherframe. That is, during a first frame f₁, a first camera exposure of afirst duration d₁ may be employed to acquire information forthree-dimensional distance measurement. Then, during a subsequent secondframe f₂, a second camera exposure of a second duration d₂ (longer thanthe duration of the first camera exposure, i.e., d₂>d₁) may be employedto acquire a two-dimensional image. During a third frame f₃ andsubsequent oddly numbered frames f₅, f₇, etc., the first duration d₁ isagain employed for the exposure to acquire additional information forthree-dimensional distance measurement. During a fourth frame f₄ andsubsequent evenly numbered frames f₆, etc., the second duration d₂ isagain employed for the exposure to acquire additional two-dimensionalimages, and so on.

In one example, a laser (or projection light source) pulse of a third,fixed duration p₁ may be emitted every other frame. In one example thethird duration p₁ is greater than the first duration d₁, but less thanthe second duration d₂ (i.e., d₁<p₁<d₂). In one example, the laser pulseis emitted at the same time as each camera exposure of the firstduration d₁ (e.g., every oddly numbered frame). Put another way, at thestart of each oddly numbered frame, a laser pulse of duration p₁ isemitted, and the camera shutter is opened for a window of duration d₁.Thus, the laser pulse may be used to project a pattern from which thedistance sensor may acquire information for three-dimensional distancemeasurement.

It can also be seen from FIG. 3 that each laser pulse of the thirdduration p₁ is associated with one camera exposure of the first durationd₁ and one camera exposure of the second duration d₂. That is, onecamera exposure of the first duration d₁ and one camera exposure of thesecond duration d₂ (in that order) occur between each pair of laserpulses of the third duration p₁. Subsequently, the images acquired forthree-dimensional distance measurement and the two-dimensional imagesmay be processed separately and differently.

FIG. 4 is a block diagram illustrating an example distance sensor 400 ofthe present disclosure. The distance sensor 400 may be used to detectthe distance d to an object 414. In one example, the distance sensor 400shares many components of the distance sensors described in U.S. patentapplication Ser. Nos. 14/920,246, 15/149,323, and 15/149,429. Forinstance, in one example, the distance sensor comprises a camera (orother image capturing device) 402, a processor 404, a controller 406,and a pattern projector 408. However, unlike the distance sensor 100 ofFIG. 1, the distance sensor 400 additionally includes a light emittingdiode (LED) 416 or other type of illumination means that emits light ina wavelength that is visible to the human eye (e.g., white).Alternatively, the emitted wavelength of the LED 416 may be the same asthe wavelength of the VCSEL 410.

In one example, the camera 402 may be a still or video camera. Thecamera 402 may be capable of capturing three-dimensional distance data.For instance, the camera 402 may include a detector that is capable ofdetecting a pattern of light that is projected onto the object 414,where the projected light is of a wavelength that is substantiallyinvisible to the human eye (e.g., infrared). The camera 402 may also becapable of capturing two-dimensional red, green, blue (RGB) images ofthe object 414. Thus, in one example, the camera 402 may be a red,green, blue infrared (RGBIR) camera. In this case, infrared lightemitted for three-dimensional distance sensing may be input only to thepixel of the camera 402 with the IR filter, while other wavelengths oflight can be recognized as color images by the pixel(s) on the RGBfilter. Because the three-dimensional distance sensing depends on theintensity of the projected pattern of light, and the two-dimensionalimaging depends on external brightness, the optimal exposure time forthe IR and RGB portions of the camera 102 will be different. The camera402 may have a fish-eye lens, and may be configured to capture imagedata of a field of view of up to 180 degrees.

The camera 402 may send captured image data to the processor 404. Theprocessor 404 may be configured to process the captured image data(e.g., three-dimensional distance data and two-dimensional image data)in order to calculate the distance to the object 414. For instance, thedistance may be calculated in accordance with the methods described inU.S. patent application Ser. Nos. 14/920,246, 15/149,323, and15/149,429.

The controller 406 may be configured to control operation of the othercomponents of the distance sensor, e.g., the operations of the camera402, the processor 404, the pattern projector 408, and the LED 416. Forinstance, the controller 406 may control the exposure time of the camera402 (e.g., the duration for which the camera's shutter is open), and thetiming with which the camera 402 captures images (including images ofthe object 414). As discussed in further detail below, the controller406 may set two separate exposure durations for the camera 402: a firstexposure duration during which an image of the object 414 is captured atthe same time that the pattern projector 408 projects a pattern onto theobject 414 (e.g., for three-dimensional distance sensing), and a secondexposure duration during which an image of the object 414 is captured ata time when the pattern projector 408 does not project a pattern ontothe object 414, but at which the LED 416 is illuminating the object 414(e.g., for two-dimensional image acquisition). In one example, thecontroller 406 may alternate between the first exposure duration and thesecond exposure duration.

The controller 406 may also control the duration for which the patternprojector 408 projects the pattern of light onto the object 414, as wellas the timing with which the pattern projector 408 projects the patternof light onto the object 414. For instance, the controller 406 maycontrol the duration of pulses emitted by a light source of the patternprojector 408, as discussed in further detail below.

The controller 406 may also control the duration for which the LED 416illuminates the object 414, as well as the timing with which the LED 416illuminates the object 414. For instance, the controller 406 may controlthe duration of pulses emitted by the LED 416, as discussed in furtherdetail below.

The pattern projector 408 may comprise various optics configured toproject the pattern of light onto the object 414. For instance, thepattern projector 408 may include a laser light source, such as avertical cavity surface emitting laser (VCSEL) 410 and a diffractiveoptical element (DOE) 412. The VCSEL 410 may be configured to emit beamsof laser light under the direction of the controller 406 (e.g., wherethe controller 406 controls the duration of the laser pulses). The DOE412 may be configured to split the beam of light projected by the VCSEL410 into a plurality of beams of light. The plurality of beams of lightmay fan or spread out, so that each beam creates a distinct point (e.g.,dot, dash, x, or the like) of light in the camera's field of view.Collectively, the distinct points of light created by the plurality ofbeams form a pattern. The distance to the object 414 may be calculatedbased on the appearance of the pattern on the object 414.

The LED 416 may comprise one or more light emitting diodes, or otherlight sources, capable of emitting light in a wavelength that is visibleto the human eye (e.g., white) under the direction of the controller 406(e.g., where the controller 406 controls the duration of the LEDpulses). Alternatively, the emitted wavelength of the LED 416 may be thesame as the wavelength of the VCSEL 410. The illumination provided bythe LED 416 may be used to acquire a two-dimensional image of the object414, as discussed in further detail below.

FIG. 5 is a flow diagram illustrating one example of a method 500 foradjusting the camera exposure of a distance sensor for three-dimensionaldepth sensing and two-dimensional image capture, according to thepresent disclosure. The method 500 may be performed, for example, by theprocessor 404 illustrated in FIG. 4. For the sake of example, the method500 is described as being performed by a processing system.

The method 500 may begin in step 502. In step 504, the processing systemmay set the exposure time of a camera of a distance sensor to a firstvalue. The first value may define a duration of the exposure (e.g., afirst window of time for which the shutter of the camera is open toacquire image data).

In step 506, the processing system may instruct an illumination source(e.g., an LED) of the distance sensor to illuminate an object in thedistance sensor's field of view. In one example, the light emitted toilluminate the object may comprise light in a wavelength that is visibleto the human eye. Alternatively, the emitted wavelength of theillumination source may be the same as the wavelength of the distancesensor's pattern projector. In one example, the instructions sent to theillumination source may include instructions regarding when to startemitting the light and for how long emit the light (e.g., the timing andduration of LED pulses).

In step 508, the processing system may instruct the camera to acquire afirst image of the object. In one example, the first image is a twodimensional image (which includes no data from projected patterns oflight). In one example, the time of exposure for the acquisition of thefirst image is therefore equal to the first value.

In step 510, the processing system may instruct the illumination sourceto stop illuminating the object. For instance, the instructions sent tothe illumination source may instruct the pattern projector to turn offan LED.

In step 512, the processing system may instruct a pattern projector(e.g., a system of optics including a laser light source and diffractiveoptical element) of the distance sensor to project a pattern of lightonto the object. In one example, the pattern of light may comprise lightthat is emitted in a wavelength that is substantially invisible to thehuman eye (e.g., infrared). In one example, the instructions sent to thepattern projector may include instructions regarding when to startprojecting the pattern of light and for how long to project the patternof light (e.g., the timing and duration of laser pulses).

In step 514, the processing system may set the exposure time of thecamera to a second value. The second value may define a duration of theexposure (e.g., a second window of time for which the shutter of thecamera is open to acquire image data). In one example, the second valueis smaller than the first value.

In step 516, the processing system may instruct the camera to acquire asecond image of the object, where the second image also includes thepattern of light projected onto the object by the pattern projector. Inone example, the time of exposure for the acquisition of the secondimage is therefore equal to the second value.

In step 518 the processing system may instruct the pattern projector tostop projecting the pattern of light onto the object. For instance, theinstructions sent to the pattern projector may instruct the patternprojector to turn off a laser.

In step 520, the processing system may determine whether to stop imagingthe object. For instance, imaging of the object may stop if sufficientdata (e.g., from the first and second images) has been acquired tocalculate the distance to the object. If the processing system concludesin step 520 that imaging should not be stopped, then the method 500 mayreturn to step 504 and proceed as described above to capture additionalimages of the object.

Alternatively, if the processing system concludes in step 520 thatimaging should be stopped, then the method 500 may proceed to step 522.In step 522, the processing system may process the first and secondimages in order to determine the distance to the object. For instance,any of the methods described in in U.S. patent application Ser. Nos.14/920,246, 15/149,323, and 15/149,429 may be used to calculate thedistance. Alternatively, the processing system may send the first andsecond images to a remote processing system for the distancecalculation.

The method 500 may end in step 524.

FIG. 6 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera (e.g., a videocamera), the distance projection for three-dimensional distancemeasurement, and the light emission for two-dimensional imageacquisition, where a first light source emits light at or near the timeof three-dimensional data acquisition and second, separate light sourceemits light at the time of two-dimensional image acquisition, andthree-dimensional distance measurement and two-dimensional imageacquisition alternate every other frame.

In particular, FIG. 6 shows seven frames, f₁-f₇, of the timing chart. Asin the example of FIG. 3, a three-dimensional distance measurement and atwo-dimensional image acquisition are performed alternately, every otherframe. That is, during a first frame f₁, a first camera exposure of afirst duration d₁ may be employed to acquire information forthree-dimensional distance measurement. Then, during a subsequent secondframe f₂, a second camera exposure of a second duration d₂ (longer thanthe duration of the first camera exposure, i.e., d₂>d₁) may be employedto acquire a two-dimensional image. During a third frame f₃ andsubsequent oddly numbered frames f₅, f₇, etc., the first duration d₁ isagain employed for the exposure to acquire additional information forthree-dimensional distance measurement. During a fourth frame f₄ andsubsequent evenly numbered frames f₆, etc., the second duration d₂ isagain employed for the exposure to acquire additional two-dimensionalimages, and so on.

As in the example of FIG. 3, a laser (or projection light source) pulseof a third, fixed duration p₁ may be emitted every other frame. In oneexample the third duration p₁ is greater than the first duration d₁, butless than the second duration d₂ (i.e., d₁<p₁<d₂). In one example, thelaser pulse is emitted at the same time each camera exposure of thefirst duration d₁ begins (e.g., each time every oddly numbered framebegins). Put another way, at the start of each oddly numbered frame, alaser pulse of duration p1 is emitted, and the camera shutter is openedfor a window of duration d₁. Thus, the laser pulse may be used toproject a pattern from which the distance sensor may acquire informationfor three-dimensional distance measurement.

It can also be seen from FIG. 6 that each laser pulse of the thirdduration p₁ is associated with one camera exposure of the first durationd₁ and one camera exposure of the second duration d₂. That is, onecamera exposure of the first duration d₁ and one camera exposure of thesecond duration d₂ (in that order) occur between each pair of laserpulses of the third duration p₁.

In one example, a light emitting diode (LED) (or illumination lightsource) pulse of a fourth, fixed duration p₂ may also be emitted,alternately with the laser pulses of the third duration p₁. In oneexample the fourth duration p₂ is the greatest of the first duration d₁,the second duration d₂, and the third duration p₁ (i.e., d₁<p₁<d₂<p₂).In one example, the LED pulses overlap frames; that is, the LED pulsesmay begin at the end of (e.g., more than halfway through) one frame andmay end near the middle of the subsequent frame. For instance, referringto FIG. 6, an LED pulse of fourth duration p₂ may begin in frame f₁,after the laser pulse of the third duration p₁ has ended. The same LEDpulse may end in the middle of the subsequent frame f₂ (during which nolaser pulse may occur). In one example, the LED pulse is emitted justbefore each camera exposure of the second duration d₂ (e.g., just beforeevery even numbered frame begins). Put another way, just before thestart of each even numbered frame, an LED pulse of duration p₂ isemitted, and the camera shutter is opened for a window of duration d₂which ends in the middle of the (even numbered) frame. Thus, the LEDpulse may be used to provide illumination with which the distance sensormay acquire a two-dimensional image of an object.

It can also be seen from FIG. 6 that each LED pulse of the fourthduration p₂ is associated with one camera exposure of the secondduration d₂ and one camera exposure of the first duration d₁. That is,one camera exposure of the second duration d₂ and one camera exposure ofthe first duration d₁ (in that order) occur between each pair of LEDpulses of the fourth duration p₂. Subsequently, the images acquired forthree-dimensional distance measurement and the two-dimensional imagesmay be processed separately and differently.

In another example, steps 508 and 516 of FIG. 5 can be modified so thatthe processing system instructs the camera to capture a first pluralityof (e.g., n) images and a second plurality of (e.g., n) images,respectively. Thus, during each pulse or emission of the illuminationsource or the pattern projection, a plurality of images may be captured.

For instance, FIG. 7 is an example timing chart illustrating therelationship between the frame rate and exposure of a distance sensorcamera (e.g., a video camera), the distance projection forthree-dimensional distance measurement, and the light emission fortwo-dimensional image acquisition, where a first light source emitslight at or near the time of three-dimensional data acquisition andsecond, separate light source emits light at the time of two-dimensionalimage acquisition, and three-dimensional distance measurement andtwo-dimensional image acquisition alternate every predetermined numberof frames.

In particular, FIG. 7 shows seven frames, f₁-f₇, of the timing chart.Unlike the examples of FIG. 3 and FIG. 6, where a three-dimensionaldistance measurement and a two-dimensional image acquisition alternateevery other frame, in FIG. 7, a three-dimensional distance measurementand a two-dimensional image acquisition alternate every predetermined(and configurable) number n of frames. In the particular exampleillustrated in FIG. 7, n=3. That is, during a first three frames f₁, f₂,and f₃, a first camera exposure of a first duration d₁ may be employedto acquire information for three-dimensional distance measurement. Then,during a subsequent three frames frame f₄, f₅, and f₆, a second cameraexposure of a second duration d₂ (longer than the duration of the firstcamera exposure, i.e., d₂>d₁) may be employed to acquire atwo-dimensional image. During a subsequent three frames starting withf₇, the first duration d₁ is again employed for the exposure to acquireadditional information for three-dimensional distance measurement, andso on.

As in the example of FIG. 3, a laser (or projection light source) pulseof a third, fixed duration p₁ may be emitted at the start of each framein which three-dimensional distance measurement data is acquired. In oneexample the third duration p₁ is greater than the first duration d₁, butless than the second duration d₂ (i.e., d₁<p₁<d₂). In one example, thelaser pulse is emitted at the same time each camera exposure of thefirst duration d₁ begins (e.g., each time a frame in a set of nsubsequent frames begins). Put another way, at the start of each framein a set of n frames designated for three-dimensional data acquisition,a laser pulse of duration p1 is emitted, and the camera shutter isopened for a window of duration d₁. Thus, the laser pulse may be used toproject a pattern from which the distance sensor may acquire informationfor three-dimensional distance measurement.

It can also be seen from FIG. 7 that each laser pulse of the thirdduration p₁ is associated with one camera exposure of the first durationd₁. That is, one camera exposure of the first duration d₁ occurs betweeneach pair of laser pulses of the third duration p₁.

In one example, a light emitting diode (LED) (or illumination lightsource) pulse of a fifth, fixed duration p₃ may also be emitted at thestart of each set of n frames in which a two-dimensional image isacquired. In one example the fifth duration p₂ is the greatest of thefirst duration d₁, the second duration d₂, the third duration p₁, andthe fourth duration p₂ (i.e., d₁<p₁<d₂<p₂<p₃). In one example, the LEDpulses overlap frames; that is, the LED pulses may begin at the end of(e.g., more than halfway through) one frame and may end near the middleof a frame n frames later. For instance, referring to FIG. 7, an LEDpulse of fifth duration p₃ may begin in frame f₃, after a laser pulse ofthe third duration p₁ has ended. The same LED pulse may end in themiddle of the frame n frames later, i.e., frame f₆. In one example, theLED pulse is emitted just before the first camera exposure of the secondduration d₂ (e.g., where n camera exposures of the second duration d₂occur in a row). Put another way, just before the start of the firstframe of n subsequent frames designated for two-dimensional imageacquisition, an LED pulse of duration p₃ is emitted, and the camerashutter is opened three times in a row for a window of duration d₂(which ends in the middle of each frame of the n frames) during theduration d₃. Thus, the LED pulse may be used to provide illuminationwith which the distance sensor may acquire a two-dimensional image of anobject.

It can also be seen from FIG. 7 that each LED pulse of the fifthduration p₃ is associated with n camera exposures of the second durationd₂. That is, n camera exposures of the second duration d₂ occur duringevery LED pulse of the fifth duration p₃. Subsequently, the imagesacquired for three-dimensional distance measurement and thetwo-dimensional images may be processed separately and differently.

FIG. 8 is a block diagram illustrating an example distance sensor 800 ofthe present disclosure. The distance sensor 800 may be used to detectthe distance d to an object 814. In one example, the distance sensor 800shares many components of the distance sensors described in U.S. patentapplication Ser. Nos. 14/920,246, 15/149,323, and 15/149,429. Forinstance, in one example, the distance sensor comprises a camera (orother image capturing device) 802, a processor 804, a controller 806,and a plurality of pattern projectors 808 ₁-808 ₂ (hereinafterindividually referred to as a “pattern projector 808” or collectivelyreferred to as “pattern projectors 808”). Thus, unlike the distancesensors 100 of FIGS. 1 and 4, the distance sensor 800 comprises morethan one pattern projector.

In one example, the camera 802 may be a still or video camera. Thecamera 802 may be capable of capturing three-dimensional distance data.For instance, the camera 802 may include a detector that is capable ofdetecting a pattern of light that is projected onto the object 814,where the projected light is of a wavelength that is substantiallyinvisible to the human eye (e.g., infrared). The camera 802 may also becapable of capturing two-dimensional red, green, blue (RGB) images ofthe object 814. Thus, in one example, the camera 802 may be a red,green, blue infrared (RGBIR) camera. In this case, infrared lightemitted for three-dimensional distance sensing may be input only to thepixel of the camera 802 with the IR filter, while other wavelengths oflight can be recognized as color images by the pixel(s) on the RGBfilter. Because the three-dimensional distance sensing depends on theintensity of the projected pattern of light, and the two-dimensionalimaging depends on external brightness, the optimal exposure time forthe IR and RGB portions of the camera 102 will be different. The camera802 may have a fish-eye lens, and may be configured to capture imagedata of a field of view of up to 180 degrees.

The camera 802 may send captured image data to the processor 804. Theprocessor 804 may be configured to process the captured image data(e.g., three-dimensional distance data and two-dimensional image data)in order to calculate the distance to the object 814. For instance, thedistance may be calculated in accordance with the methods described inU.S. patent application Ser. Nos. 14/920,246, 15/149,323, and15/149,429.

The controller 806 may be configured to control operation of the othercomponents of the distance sensor, e.g., the operations of the camera802, the processor 804, the pattern projectors 808, and the LED 816. Forinstance, the controller 806 may control the exposure time of the camera802 (e.g., the duration for which the camera's shutter is open), and thetiming with which the camera 802 captures images (including images ofthe object 814). As discussed in further detail below, the controller806 may set two separate exposure durations for the camera 802: a firstexposure duration during which an image of the object 814 is captured atthe same time that at least one of the pattern projectors 808 projects apattern onto the object 814 (e.g., for three-dimensional distancesensing), and a second exposure duration during which an image of theobject 814 is captured at a time when the pattern projectors 808 do notproject a pattern onto the object 814, but at which the LED 816 isilluminating the object 814 (e.g., for two-dimensional imageacquisition). In one example, the controller 806 may alternate betweenthe first exposure duration and the second exposure duration.

The controller 806 may also control the duration for which the patternprojectors 808 project the pattern of light onto the object 814, as wellas the timing with which the pattern projectors 808 project the patternof light onto the object 814. For instance, the controller 806 maycontrol the duration of pulses emitted by a light source of the patternprojectors 808, as discussed in further detail below. In one particularexample, the controller 806 may control the pattern projectors 808 toproject the pattern of light into separate portions of the camera'sfield of view at separate times.

The controller 806 may also control the duration for which the LED 816illuminates the object 814, as well as the timing with which the LED 816illuminates the object 814. For instance, the controller 806 may controlthe duration of pulses emitted by the LED 816, as discussed in furtherdetail below.

The pattern projectors 808 may comprise various optics configured toproject the pattern of light onto the object 814. For instance, eachpattern projector 808 may include a respective laser light source, suchas a respective vertical cavity surface emitting laser (VCSEL) 810 ₁ or810 ₂ (hereinafter also referred to individually as a “VCSEL 810” orcollectively as “VCSELs 810”) and a respective diffractive opticalelement (DOE) 812 ₁ or 812 ₂ (hereinafter referred to individually as a“DOE 812” or collectively as “DOEs 812”). The VCSELs 810 may beconfigured to emit beams of laser light under the direction of thecontroller 806 (e.g., where the controller 806 controls the duration ofthe laser pulses). The DOEs 812 may be configured to split the beams oflight projected by the respective VCSELs 810 into respective pluralitiesof beams of light. The pluralities of beams of light may fan or spreadout, so that each beam creates a distinct point (e.g., dot, dash, x, orthe like) of light in the camera's field of view. Collectively, thedistinct points of light created by the pluralities of beams formrespective patterns. The distance to the object 814 may be calculatedbased on the appearance of the patterns on the object 814.

The LED 816 may comprise one or more light emitting diodes, or otherlight sources, capable of emitting light in a wavelength that is visibleto the human eye (e.g., white) under the direction of the controller 806(e.g., where the controller 806 controls the duration of the LEDpulses). Alternatively, the emitted wavelength of the LED 816 may be thesame as the wavelength of the VCSEL 810. The illumination provided bythe LED 816 may be used to acquire a two-dimensional image of the object814, as discussed in further detail below.

FIG. 9 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera (e.g., a videocamera), the distance projection for three-dimensional distancemeasurement, and the light emission for two-dimensional imageacquisition, where two light projection systems (e.g., used forthree-dimensional distance data acquisition) are used and the exposuredurations for three-dimensional data acquisition and two-dimensionalimage capture are the same.

In particular, FIG. 9 shows seven frames, f₁-f₇, of the timing chart. Inone example, two three-dimensional distance measurement and onetwo-dimensional image acquisition are performed alternately, every threeframe. That is, during a first frame f₁, a first camera exposure of afirst duration d₁ may be employed to acquire information forthree-dimensional distance measurement. During a subsequent second framef₂, a second camera exposure of the first duration d₁ may be employed toacquire information for three-dimensional distance measurement. Then,during a subsequent third frame f₃, a third camera exposure of the firstduration d₁ may be employed to acquire a two-dimensional image. During afourth frame f₄ and a fifth frame f₅, the first duration d₁ is againemployed for the exposure to acquire additional information forthree-dimensional distance measurement. During a sixth frame f₆, thefirst duration d₁ is again employed for the exposure to acquireadditional two-dimensional images, and so on.

As illustrated in FIG. 9, a first laser (or projection light source)pulse of a third, fixed duration p₁ may be emitted every three frames.In one example the third duration p₁ is greater than the first durationd₁ (i.e., d₁<p₁). In one example, the first laser pulse is emitted atthe beginning of every third frame. Put another way, at the start ofevery third frame, a first laser pulse of duration p1 is emitted, andthe camera shutter is opened for a window of duration d₁. Thus, thefirst laser pulse may be used to project a pattern from which thedistance sensor may acquire information for three-dimensional distancemeasurement.

Similarly, a second laser (or projection light source) pulse of thethird, fixed duration p₁ may also be emitted every three frames. In oneexample, the second laser pulse is emitted at the beginning of everythird frame, but one frame after each first laser pulse. Put anotherway, at the start of every frame after a frame in which the first laserpulse occurs, a second laser pulse of duration p1 is emitted, and thecamera shutter is opened for a window of duration d₁. Thus, the secondlaser pulse may be used to project a pattern from which the distancesensor may acquire information for three-dimensional distancemeasurement.

It can also be seen from FIG. 9 that each laser pulse (whether it is afirst laser pulse or a second laser pulse) of the third duration p₁ isassociated with three camera exposures of the first duration d₁. Thatis, three camera exposures of the first duration d₁ occur between eachpair of (first or second) laser pulses of the third duration p₁.

In one example, a light emitting diode (LED) (or illumination lightsource) pulse of a fourth, fixed duration p₂ may also be emitted, aftereach second laser pulse of the third duration p₁. In one example thefourth duration p₂ is the greatest of the first duration d₁ and thethird duration p₁ (i.e., d₁<p₁<p₂). In one example, the LED pulsesoverlap frames; that is, the LED pulses may begin at the end of (e.g.,more than halfway through) one frame and may end near the middle of thesubsequent frame. For instance, referring to FIG. 9, an LED pulse offourth duration p₂ may begin in frame f₂, after the second laser pulseof the third duration p₁ has ended. The same LED pulse may end in themiddle of the subsequent frame f₃ (during which no laser pulse mayoccur). In one example, the LED pulse is emitted just before each cameraexposure during which two-dimensional image acquisition is performed(e.g., just before every third frame begins). Put another way, justbefore the start of every third frame, an LED pulse of duration p₂ isemitted, and the camera shutter is opened for a window of duration d₁which ends in the middle of the subsequent frame. Thus, the LED pulsemay be used to provide illumination with which the distance sensor mayacquire a two-dimensional image of an object.

It can also be seen from FIG. 9 that each LED pulse of the fourthduration p₂ is associated with three camera exposures of the firstduration d₁. That is, three camera exposures of the first duration d₁occur between each pair of LED pulses of the fourth duration p₂.Subsequently, the images acquired for three-dimensional distancemeasurement and the two-dimensional images may be processed separatelyand differently.

FIG. 10 is a flow diagram illustrating one example of a method 1000 foradjusting the camera exposure of a distance sensor for three-dimensionaldepth sensing and two-dimensional image capture, according to thepresent disclosure. The method 1000 may be performed, for example, byany of the processors 104, 404, or 804 illustrated in FIGS. 1, 4, and 8.However, in this case, the processor additionally performs an analysisof two-dimensional image data captured by the camera and feeds thisanalysis back into the controller to control the exposure control andillumination control. For the sake of example, the method 1000 isdescribed as being performed by a processing system.

The method 1000 may begin in step 1002. In step 1004, the processingsystem may set the exposure time of a camera of a distance sensor to afirst value. The first value may define a duration of the exposure(e.g., a first window of time for which the shutter of the camera isopen to acquire image data).

In step 1006, the processing system may instruct an illumination source(e.g., an LED) of the distance sensor to illuminate an object in thedistance sensor's field of view. In one example, the light emitted toilluminate the object may comprise light in a wavelength that is visibleto the human eye (e.g., white). In one example, the instructions sent tothe illumination source may include instructions regarding when to startemitting the light and for how long emit the light (e.g., the timing andduration of LED pulses).

In step 1008, the processing system may instruct the camera to acquire afirst image of the object. In one example, the first image is a twodimensional image (which includes no data from projected patterns oflight). In one example, the time of exposure for the acquisition of thefirst image is therefore equal to the first value.

In step 1010, the processing system may instruct the illumination sourceto stop illuminating the object. For instance, the instructions sent tothe illumination source may instruct the pattern projector to turn offan LED.

In step 1012, the processing system may determine a second value for theexposure time of the camera and a projection time of a pattern projectorof the distance sensor (e.g., a system of optics including a laser lightsource and diffractive optical element), based on an analysis of thefirst image of the object.

In step 1014, the processing system may instruct the pattern projectorof the distance sensor to project a pattern of light onto the object. Inone example, the pattern of light may comprise light that is emitted ina wavelength that is substantially invisible to the human eye (e.g.,infrared). In one example, the instructions sent to the patternprojector may include instructions regarding when to start projectingthe pattern of light and for how long to project the pattern of light(e.g., the timing and duration of laser pulses).

In step 1016, the processing system may set the exposure time of thecamera to the second value. The second value may define a duration ofthe exposure (e.g., a second window of time for which the shutter of thecamera is open to acquire image data). In one example, the second valueis smaller than the first value.

In step 1018, the processing system may instruct the camera to acquire asecond image of the object, where the second image also includes thepattern of light projected onto the object by the pattern projector. Inone example, the time of exposure for the acquisition of the secondimage is therefore equal to the second value.

In step 1020 the processing system may instruct the pattern projector tostop projecting the pattern of light onto the object. For instance, theinstructions sent to the pattern projector may instruct the patternprojector to turn off a laser.

In step 1022, the processing system may determine whether to stopimaging the object. For instance, imaging of the object may stop ifsufficient data (e.g., from the first and second images) has beenacquired to calculate the distance to the object. If the processingsystem concludes in step 1022 that imaging should not be stopped, thenthe method 1000 may return to step 1004 and proceed as described aboveto capture additional images of the object.

Alternatively, if the processing system concludes in step 1022 thatimaging should be stopped, then the method 1000 may proceed to step1024. In step 1024, the processing system may process the first andsecond images in order to determine the distance to the object. Forinstance, any of the methods described in in U.S. patent applicationSer. Nos. 14/920,246, 15/149,323, and 15/149,429 may be used tocalculate the distance. Alternatively, the processing system may sendthe first and second images to a remote processing system for thedistance calculation.

The method 1000 may end in step 1026.

FIG. 11 is an example timing chart illustrating the relationship betweenthe frame rate and exposure of a distance sensor camera (e.g., a videocamera), the distance projection for three-dimensional distancemeasurement, and the light emission for two-dimensional imageacquisition, where information about shutter speed at the time oftwo-dimensional image acquisition is fed back to the timing forthree-dimensional distance data acquisition. That is, the exposure timeof the camera and the time of projection for a pattern of light duringthree-dimensional distance data acquisition may be based on an analysisof the object from a two-dimensional image of the object.

It should be noted that although not explicitly specified, some of theblocks, functions, or operations of the methods 200, 500, and 1000described above may include storing, displaying and/or outputting for aparticular application. In other words, any data, records, fields,and/or intermediate results discussed in the methods 200, 500, and 1000can be stored, displayed, and/or outputted to another device dependingon the particular application. Furthermore, blocks, functions, oroperations in FIGS. 2, 5, and 10 that recite a determining operation, orinvolve a decision, do not imply that both branches of the determiningoperation are practiced. In other words, one of the branches of thedetermining operation may not be performed, depending on the results ofthe determining operation.

FIG. 12 depicts a high-level block diagram of an example electronicdevice 1100 for calculating the distance from a sensor to an object. Assuch, the electronic device 2300 may be implemented as a processor of anelectronic device or system, such as a distance sensor (e.g., asprocessor 104, 404, or 804 in FIGS. 1, 4, and 8).

As depicted in FIG. 12, the electronic device 1200 comprises a hardwareprocessor element 1202, e.g., a central processing unit (CPU), amicroprocessor, or a multi-core processor, a memory 1204, e.g., randomaccess memory (RAM) and/or read only memory (ROM), a module 1205 forcalculating the distance from a sensor to an object, and variousinput/output devices 1206, e.g., storage devices, including but notlimited to, a tape drive, a floppy drive, a hard disk drive or a compactdisk drive, a receiver, a transmitter, a display, an output port, aninput port, and a user input device, such as a keyboard, a keypad, amouse, a microphone, a camera, a laser light source, an LED lightsource, and the like.

Although one processor element is shown, it should be noted that theelectronic device 1200 may employ a plurality of processor elements.Furthermore, although one electronic device 1200 is shown in the figure,if the method(s) as discussed above is implemented in a distributed orparallel manner for a particular illustrative example, i.e., the blocksof the above method(s) or the entire method(s) are implemented acrossmultiple or parallel electronic devices, then the electronic device 1200of this figure is intended to represent each of those multipleelectronic devices.

It should be noted that the present disclosure can be implemented bymachine readable instructions and/or in a combination of machinereadable instructions and hardware, e.g., using application specificintegrated circuits (ASIC), a programmable logic array (PLA), includinga field-programmable gate array (FPGA), or a state machine deployed on ahardware device, a general purpose computer or any other hardwareequivalents, e.g., computer readable instructions pertaining to themethod(s) discussed above can be used to configure a hardware processorto perform the blocks, functions and/or operations of the abovedisclosed method(s).

In one example, instructions and data for the present module or process1205 for calculating the distance from a sensor to an object, e.g.,machine readable instructions can be loaded into memory 1204 andexecuted by hardware processor element 1202 to implement the blocks,functions or operations as discussed above in connection with themethods 200, 500, and 1000. Furthermore, when a hardware processorexecutes instructions to perform “operations”, this could include thehardware processor performing the operations directly and/orfacilitating, directing, or cooperating with another hardware device orcomponent, e.g., a co-processor and the like, to perform the operations.

The processor executing the machine readable instructions relating tothe above described method(s) can be perceived as a programmed processoror a specialized processor. As such, the present module 1205 forcalculating the distance from a sensor to an object of the presentdisclosure can be stored on a tangible or physical (broadlynon-transitory) computer-readable storage device or medium, e.g.,volatile memory, non-volatile memory, ROM memory, RAM memory, magneticor optical drive, device or diskette and the like. More specifically,the computer-readable storage device may comprise any physical devicesthat provide the ability to store information such as data and/orinstructions to be accessed by a processor or an electronic device suchas a computer or a controller of a safety sensor system.

It will be appreciated that variants of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intomany other different systems or applications. Various presentlyunforeseen or unanticipated alternatives, modifications, or variationstherein may be subsequently made which are also intended to beencompassed by the following claims.

What is claimed is:
 1. A method to acquire three-dimensional depth byprojecting a pattern of light, comprising: setting, by a processingsystem of a distance sensor, an exposure time of a camera of thedistance sensor to a first value; instructing, by the processing system,the camera to acquire a first image of an object in a field of view ofthe camera without projecting the pattern of light, where the firstimage is acquired while the exposure time is set to the first value;instructing, by the processing system, a pattern projector of thedistance sensor to project the pattern of light onto the object;setting, by the processing system, the exposure time of the camera to asecond value that is different than the first value; and instructing, bythe processing system, the camera to acquire a second image of theobject, where the second image includes the pattern of light, and wherethe second image is acquired while the exposure time is set to thesecond value.
 2. The method of claim 1, wherein the second value issmaller than the first value.
 3. The method of claim 2, wherein thefirst image is a two-dimensional red, green, blue image.
 4. The methodof claim 3, wherein the second image is an infrared image.
 5. The methodof claim 1, wherein the pattern of light comprises a pattern of lightwhose wavelength is invisible to a human eye.
 6. The method of claim 5,wherein the wavelength is infrared.
 7. The method of claim 5, whereinthe camera comprises a detector that detects wavelengths of light thatare visible to the human eye as well as the wavelength that is invisibleto the human eye.
 8. The method of claim 7, wherein the camera is a red,green, blue infrared camera.
 9. The method of claim 1, furthercomprising: instructing, by the processing system and prior to theinstructing the camera to acquire the first image, an illuminationsource of the distance sensor to illuminate the object; and instructing,by the processing system and prior to the instructing the patternprojector to project the pattern of light, the illumination source tostop illuminating the object.
 10. The method of claim 9, wherein theillumination source emits light in a wavelength that is invisible to ahuman eye.
 11. The method of claim 10, wherein the illumination sourceis an infrared light emitting diode.
 12. The method of claim 1, whereinthe first image is one of a first plurality of images acquired by thecamera while the exposure time is set to the first value, and whereinthe second image is one of a second plurality of images acquired by thecamera while the exposure time is set to the second value.
 13. Themethod of claim 1, wherein the second value is determined based at leastin part on an analysis of the first image.
 14. The method of claim 1,wherein the pattern projector is a first pattern projector of aplurality of pattern projectors of the distance sensor, and wherein eachpattern projector of the plurality of pattern projectors is configuredto project a respective pattern of light into a different area in thefield of view.
 15. The method of claim 1, further comprising:calculating, by the processing system, three-dimensional informationincluding a distance from the distance sensor to the object, based on ananalysis of the first image and the second image.
 16. A non-transitorymachine-readable storage medium encoded with instructions executable bya processor of a distance sensor to project a pattern of light, wherein,when executed, the instructions cause the processor to performoperations, the operations comprising: setting an exposure time of acamera of the distance sensor to a first value; instructing the camerato acquire a first image of an object in a field of view of the camera,where the first image is acquired while the exposure time is set to thefirst value and without projecting the pattern of light; instructing apattern projector of the distance sensor to project the pattern of lightonto the object; setting the exposure time of the camera to a secondvalue that is different than the first value; instructing the camera toacquire a second image of the object, where the second image includesthe pattern of light, and where the second image is acquired while theexposure time is set to the second value; and calculatingthree-dimensional information including a distance from the distancesensor to the object, based on an analysis of the first image and thesecond image.
 17. A distance sensor, comprising: a processor; and anon-transitory machine-readable storage medium encoded with instructionsexecutable by the processor to project a pattern of light, wherein, whenexecuted, the instructions cause the processor to perform operations,the operations comprising: setting an exposure time of a camera of thedistance sensor to a first value; instructing the camera to acquire afirst image of an object in a field of view of the camera, where thefirst image is acquired while the exposure time is set to the firstvalue and without projecting the pattern of light; instructing a patternprojector of the distance sensor to project the pattern of light ontothe object; setting the exposure time of the camera to a second valuethat is different than the first value; instructing the camera toacquire a second image of the object, where the second image includesthe pattern of light, and where the second image is acquired while theexposure time is set to the second value; and calculatingthree-dimensional information including a distance from the distancesensor to the object, based on an analysis of the first image and thesecond image.
 18. The distance sensor of claim 17, wherein the patternprojector comprises: an infrared laser source configured to emit a beamof infrared light; and a diffractive optical element configured to splitthe beam of infrared light into a plurality of beams of infrared light,wherein each beam of the plurality of beams of infrared light createsone projection artifact in a field of view of the camera.
 19. Thedistance sensor of claim 18, wherein the camera is a red, green, blueinfrared camera.
 20. The distance sensor of claim 17, furthercomprising: an infrared light emitting diode controlled by the processorto illuminate the object when the camera captures the first image.